Information processing system

ABSTRACT

In an information processing system provided with a camera and a microphone, for transmitting and receiving information of a user to and from another information processing system through a transmission lime, image data of the user obtained by a camera is stored in a memory in advance. When one user communicates another user, image data of the one user is obtained by the camera and is synthesized with the image data stored in the memory in advance by image-processing. The clothes, hair, background, make-up etc. of the one user are made different to reality and the image data are transmitted to the another user in the communication.

This application is a Divisional of application Ser. No. 08/505,947,filed Jul. 24, 1995.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing system whichis provided with a camera and a microphone and transmits/receives imagedata and voice (sound) data of the users to/from another informationprocessing system through a transmission line.

2. Description of the Related Art

There have been known information processing systems for transmittingand receiving image data and voice data via a communication line using,as terminals, information processing systems provided with a camera anda microphone, such as communication using videophones or computers.

With these information processing systems, not only voice can betransmitted, but the conversing parties can also talk while looking ateach other's face expression, and further images such as characters orphotographs can also be transmitted, and it is therefore possible totransmit and receive a larger volume of data than with voice only, andthus detailed and profound conversations can be held.

With these information processing systems, image data of both of thecommunicating parties are transmitted, regardless of the intention ofthe conversing parties. Thus, before the parties involved in thecommunication communicate with each other, it is often necessary toperform operations which are not necessary for the transmission, such asarranging one's appearance and tidying the surroundings.

Simplicity of transmission is therefore lost. Further, there is adeficiency in terms of maintaining the privacy of the receiving partysince, in comparison with the calling party who has, in advance, beenintending to have a conversation, the receiving party may not have madesufficient preparation for the conversation.

In other words, with conventional data communication involving voice andimages, the appearance of the users is transmitted directly as imagedata to the other party, and therefore if it is necessary to communicatewhen the receiving party of the communication does not wish to do so,the receiving party is subjected to psychological anguish.

If the receiving party is compelled, having just woken up, tocommunicate using an information processing system provided with acamera and a microphone, that is, if, clothes, hair or make-up, thesurroundings and the like are in disarray, such that the receiving partywould be impolite to the other party to appear, or if the receivingparty were extremely tired and the expression was likely to becomegloomy, the receiving party would be forced to refuse to send to theother party the image data of the receiving party, which had been takenby the camera, and thus to refuse to transmit image data.

Thus to protect the minimal privacy of the receiving party of thecommunication, transmission of image data is stopped and communicationof only voice data is performed. Therefore the services usinginformation communication of image and voice are not effectivelyutilized.

Further, if there is a sudden need for the calling party to transmitinformation, then even the receiving party may decide to refrain fromeffecting communication that is accompanied by the transmission ofimages, unless the receiving party has arranged so as not to appearimpolite to the other party, and tidied the surroundings, and thus inthe end there is a likelihood that the matter will be dealt with using anormal voice-only telephone or a facsimile.

SUMMARY OF THE INVENTION

The object of the present invention is to provide an image processingsystem whereby, when image data and voice data are being transmitted andreceived using an information processing system provided with a cameraand a microphone, conversation is possible using images and voice(sound) without the transmission of image data which the transmittingparty does not wish to be transmitted.

In order to solve the above problem, one of the main inventionsdisclosed in the present specification is an information processingsystem for transmitting and receiving information of a user to and fromanother information processing system through a transmission line, thesystem comprising, a camera for obtaining image data of the user, amicrophone for obtaining voice (sound) data of the user, means forimage-processing the image data, to obtain different image data inportions other than a face of the user, and means for transmitting theimage-processed image data and the obtained voice data to the anotherinformation processing system. In the construction, portions other thanthe face correspond to the background, clothes, hair, portions below theneck and the like.

Further, one important construction disclosed in the presentspecification is An information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of theuser, a microphone for obtaining voice data of the user, means forchanging the one image data for another image data or synthesizing theone image data with another image data, and means for transmitting thechanged or synthesized image data and the voice data to the anotherinformation processing system.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of theuser and another image data representing a movement of a mouth of theuser, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for synthesizing the storedone image data with the another image data, and means for transmittingthe synthesizing image data and the voice data to the anotherinformation processing system.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of a faceof the user and another image data representing a movement of a mouth ofthe user, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for detecting a degree oftiredness of the user by image-processing the stored one image data, andchanging the stored one image data for synthesizing image data inaccordance with the degree of tiredness of the user, wherein thesynthesizing image data is obtained by synthesizing the stored one imagedata with the another image data, and means for transmitting the voicedata and one of the stored one image data and the synthesizing imagedata to the another information processing system.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of a faceof the user and another image data representing a movement of a mouth ofthe user, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for detecting a degree oftiredness of the user by image-processing the stored one image data andchanging the stored one image data for synthesizing image data inaccordance with the degree of tiredness of the user, wherein thesynthesizing image data is obtained by synthesizing the stored one imagedata with the another image data, and means for transmitting the voicedata and one of the stored one image data and the synthesizing imagedata to the another information processing system, wherein the degree oftiredness of the user is obtained by measuring the number of blinks ofthe user per unit time in accordance with the one image data, andcomparing the number of the blinks of the user with a reference blinkvalue representing the reference number of blinks of the user per unittime, measured in advance.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of a faceof the user and another image data representing a movement of a mouth ofthe user, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for detecting a degree oftiredness of the user by image-processing the stored one image data, andchanging the stored one image data for synthesizing image data inaccordance with the degree of tiredness of the user, wherein thesynthesizing image data is obtained by synthesizing the stored one imagedata with the another image data, and means for transmitting the voicedata and one of the stored one image dada and the synthesizing imagedata to the another information processing system, wherein the degree oftiredness of the user is obtained by measuring the number of blinks ofthe user per unit time and identifying the user in accordance with theone image data, referring to a reference blink value of the user inaccordance with the identification, and comparing the number of theblinks of the user with the reference blink value of the user, thereference blink value representing the reference number of blinks of theuser per unit time, measured in advance.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of a faceof the user and another image data representing a movement of a mouth ofthe user, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for detecting a degree oftiredness of the user by image-processing the stored one image data, andchanging the stored one image data for synthesizing image data inaccordance with the degree of tiredness of the user, wherein thesynthesizing image data is obtained by synthesizing the stored one imagedata with the another image data, and means for transmitting the voicedata and one of the stored one image data and the synthesizing imagedata to the another information processing system, wherein the degree oftiredness of the user is obtained by measuring the number of blinks ofthe user per unit time and identifying the user with an ID number,referring to a reference blink value of the user in accordance with theidentification, and comparing the number of the blinks of the user withthe reference blink value of the user, the reference blink valuerepresenting the reference number of blinks of the user per unit time,measured in advance.

To elaborate, one of the important inventions disclosed by the presentspecification is an information processing system (for example, apersonal computer or videophone connected to a transmission line)provided with a camera and a microphone, wherein the number of blinks ofthe user is obtained from image data taken in by the camera, and thedegree of tiredness of the user is estimated by calculating the numberof blinks per unit time using an internal timing circuit; if thisexceeds a certain fixed value, the actual image data of the user are nottransmitted but image data which are a combination of the movement ofthe mouth of the user with image data which have been taken and storedin advance are transmitted.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of a faceof the user and another image data representing a movement of a mouth ofthe user, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for detecting a degree oftiredness of the user by image-processing the stored one image data, andchanging the stored one image data for synthesizing image data inaccordance with the degree of tiredness of the user, wherein thesynthesizing image data is obtained by synthesizing the stored one imagedata with the another image data, and means for transmitting the voicedata and one of the stored one image data and the synthesizing imagedata to the another information processing system, wherein the degree oftiredness of the user is obtained by measuring a bloodshot degree ofeyeballs of the user in accordance with the one image data.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of a faceof the user and another image data representing a movement of a mouth ofthe user, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for detecting a degree oftiredness of the user by image-processing the stored one image data, andchanging the stored one image data for synthesizing image data inaccordance with the degree of tiredness of the user, wherein thesynthesizing image data is obtained by synthesizing the stored one imagedata with the another image data, and means for transmitting the voicedata and one of the stored one image data and the synthesizing imagedata to the another information processing system, wherein the degree oftiredness of the user is obtained by measuring a current bloodshotdegree of eyeballs of the user and identifying the user in accordancewith the one image data, referring to reference image data having areference bloodshot degree in accordance with the identification, andcomparing the current bloodshot degree of the eyeballs of the user withthe reference bloodshot degree.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining one image data representing an expression of a faceof the user and another image data representing a movement of a mouth ofthe user, a microphone for obtaining voice data of the user, means forstoring the one image data in advance, means for detecting a degree oftiredness of the user by image-processing the stored one image data, andchanging the stored one image data for synthesizing image data inaccordance with the degree of tiredness of the user, wherein thesynthesizing image data is obtained by synthesizing the stored one imagedata with the another image data, and means for transmitting the voicedata and one of the stored one image data and the synthesizing imagedata to the another information processing system wherein the degree oftiredness of the user is obtained by measuring a current bloodshotdegree of eyeballs of the user and identifying the user with an IDnumber, referring to reference image data having a reference bloodshotdegree in accordance with the identification, and comparing the currentbloodshot degree of the eyeballs of the user with the referencebloodshot degree.

In other words, one of the important inventions disclosed in the presentspecification is an information Processing system provided with a cameraand a microphone, wherein the degree of tiredness of the user isestimated by obtaining the bloodshot degree of the eyeballs of the userfrom image data taken in by the camera, and wherein if this exceeds acertain fixed value, the actual image data of the user are nottransmitted but image data which are a combination of the movement ofthe mouth of the user and image data which have been taken and stored inadvance are transmitted.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising acamera for obtaining image data of the user including background data, amicrophone for obtaining voice data of the user, means for storingdesired background data, means for changing the background data for thedesired background data by image processing, and means for transmittingthe image-processed image data and the voice data to the anotherinformation processing system.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining image data including a face of the user, amicrophone for obtaining voice data of the user, means for making up theface of the user included in the image data by image processing, andmeans for transmitting the image-processed image data and the voice datato the another information processing system.

Further, one important construction disclosed in the presentspecification is an information processing system for transmitting andreceiving information of a user to and from another informationprocessing system through a transmission line, the system comprising, acamera for obtaining image data of the user including face data andbackground data, a microphone for obtaining voice data of the user,means for storing desired background data, means for making up the facedata of the image data and changing the background data for the desiredbackground data by image processing, and means for transmitting theimage-processed image data and the voice data to the another informationprocessing system.

In other words, another important invention disclosed by the presentspecification is an image processing system provided with a camera and amicrophone, wherein the background is changed arbitrarily by theselection of the user, and the image data of the background of the userwhich is to be transmitted to another information processing system ischanged.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an information processing system whichis provided with a camera and a microphone and performs imageprocessing;

FIG. 2 shows the principle of pattern matching in image processing;

FIG. 3 shows an example of the external construction of an informationprocessing system;

FIGS. 4 to 6 show a flowchart for image processing.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 shows a block diagram of an information processing system of thepresent embodiment, and FIG. 3 shows the external construction of theinformation processing system. This is an example of a PC (personalcomputer) or WS (workstation) which is provided with a CCD (chargecoupled device) camera and a microphone.

The embodiment information processing system includes a micro processingunit (MPU) 101, a main memory 102, a memory controller 103, a videorandom access memory (VRAM) 104, a multiplexer 105, a digital signalprocessor (DSP) 106, CCD camera 107, an A/D converter 108, a microphone109, a line large scale integrated (LSI) unit 110, a speaker 111, a D/Aconverter 112, a display unit 113, a display unit controller 114, akeyboard 115, a keyboard controller 116, a pointing device 117, apointing device controller 118, a flash memory 119, a hard disk 120 anda hard disk controller 121.

The MPU 101 controls the entire system, such as controls of the memorycontroller 103, the line LSI unit 110 and the like.

The main memory 102 is used for image-processing the image datarepresenting the face of a user and the like obtained by the CCD camera107. That is, image processing is performed in the main memory 102 bythe MPU 101 to detect the degree of tiredness from the expression of theuser included in the image data. The memory controller 103 performs therefresh operation and the read/write operation control for the VRAM 104,and transfers data stored in the VRAM 104 to the main memory 102 bydirect memory access (DMA) transfer. In the VRAM 104, the image data andthe voice (sound) data of a user are stored.

The multiplexer 105 selects image data obtained by the CDD camera 107 orthe voice data obtained by the microphone.

The CCD camera 107 obtains the image data of a user. The obtained imagedata (video signal) is binarized by the digital signal processor 106 andthen is stored in the VRAM 104.

The microphone 109 obtains the voice of a user as a voice signal. TheA/D convertor 108 digitizes the voice signal. The voice data is storedin the VRAM 104 through the multiplexer 105.

The line LSI unit 110 operates as an interface for an external line andtransmits the image data and the voice data by DMA transfer to thedestination using a packet format defined by the protocol of theexternal line. Also, the unit 110 receives image data and voice data ofanother user from the destination.

In the present embodiment, with an intelligent option board or a dumboption board connected to the bus of the PC or WS, it is also possibleto control the peripheral circuits directly from the CPU (centralprocessing unit) of the PC or WS.

The speaker 111 mainly outputs, through the D/A converter 112, the voicedata of another user transmitted from another information processingsystem, to regenerate the voice of the transmitting user as the anotheruser.

The display unit 113 mainly displays image data of another usertransmitted from another information processing system, under control ofthe display unit controller 114.

The keyboard 115 is controlled by the keyboard controller 116 for keyinput control and used to key-input various information such as IDinformation of users.

The pointing device 117 includes a mouse, a ball or the like and iscontrolled by the pointing device controller 118.

In such information processing system, the image data obtained by theCCD camera 107 is image-processed if necessary and then transmitted toanother information processing system.

An operation of the information processing system will be described withFIGS. 4 to 6.

To prepare the communication to the user of another informationprocessing system, image data representing a face of a user and the likeand voice data is obtained in the present system in advance. That is,the image data (original image data) of the user is obtained by the CDDcamera 107 (step A1). The obtained image data (video signal) isbinarized by the digital signal processor 106 and then stored in theVRAM 104 (step A3). Also, at the same time, voice data (original voicedata) of the user is obtained by the microphone 109 (step A2). Theobtained voice data (analog signal) is digitized by the AD converter 108and then stored in the VRAM 104 (step A3).

When the communication is requested by another user of anotherinformation processing system, the image data of the user and the likein the present system is transmitted by the following operation.

First, a plurality of current image data of the user are obtained by theCCD camera 107 during a desired period of time (step B1) and then storedin the VRAM 104 through the digital signal processor 106 and themultiplexer 105. In the MPU 101, the following processes are performedin accordance with an image-processing program to be stored in advance.That is, in step B2, an eye portion of the user is extracted from theobtained current image data.

In step B3, it is determined whether an edge of the extracted eyeportion is detected or not. If the edge is detected, it can bedetermined that the eyes were closed, that is, the user blinked. Thedetermination is performed for each extracted eye portion, and thenumber of eyewinks (blinks) of the user is counted (step B4).

In step B5, it is determined that a count value counted by the step B4is greater than or equal to a desired value. That is, it is determinedwhether or not the number of blinks of the user during a desired time isno less than the desired number of blinks. This is based on the matterthat there is the number of blinks as a parameter representing thedegree of tiredness of a human and the number of blinks increases inaccordance with the increase of the degree of the tiredness.

In step B5, when the count value is greater than or equal to the desiredvalue, it is determined that the user is tired. Therefore, the originalimage data of the user stored in the step A3 in advance is read out fromthe VRAM 104, and the current image data is synthesized with theoriginal image data (step B6). The synthesized image data is transmittedto the destination of another information processing system through theline LSI unit 110 (step B7).

On the other hand, in step B5, when the count value is smaller than thedesired value, it is determined that the user is not tired. Therefore,the current image data is transmitted to the destination of anotherinformation processing system through the line LSI unit 110 withoutimage synthesizing (step B8).

The above image-processing is performed in th main memory 102 by the MPU101. At this time, the number of blinks of the user during a desiredperiod is calculated by the MPU 101. Then, when the number of blinks isno less than a desired value, an image of the user stored in advance inthe hard disk 120, the flash memory 119 or the like is synthesized withthe mouth portion of the current image of the user. The synthesizedimage is DMA-transferred by the line LSI unit 110, thereby to transmitthe image data to the transmission line. Therefore it is possible totransmit image data just as though the user is speaking with the clothesand the background stored in advance.

As a parameter representing the degree of tiredness, other than thenumber of blinks, the bloodshot degree can be used. In this case, forexample, the following processing is performed in the present system.

First, a plurality of current image data of the user are obtained by theCCD camera 107 during a desired period of time (step C1) and then storedin the VRAM 104 through the digital signal processor 106 and themultiplexer 105. In the MPU 101, the following processes are performedin accordance with an image-processing program to be stored in advance.That is, in step C2, an eye portion of the user is extracted from theobtained current image data.

In step C3, a bloodshot portion is detected from the extracted eyeportion. Since the bloodshot portion may be red, when the degree of redin a portion is greater than a desired level, it is determined that theportion is a bloodshot portion, and the portion is detected as thebloodshot portion. Further, an area ratio of the bloodshot portion tothe eye portion is measured (step C4). The area ratio is used as anindex for determining the bloodshot degree of the user.

In step C5, it is determined that the area ratio of the bloodshotportion is greater than or equal to a desired value. When the area ratioof the bloodshot portion is no less than the desired value, it isdetermined that the user is tired. Therefore, the original image data ofthe user stored in the step A3 in advance is read out from the VRAM 104,and the current image data is synthesized with the original image data(step B6). The synthesized image data is transmitted to the destinationof another information processing system through the line LSI unit 110(step B7).

On the other hand, in step B5, when the area ratio of the bloodshotportion is smaller than the desired value, it is determined that theuser is not tired. Therefore, the current image data is transmitted tothe destination of another information processing system through theline LSI unit 110 without image synthesizing (step B8).

Therefore it is possible to transmit image data just as though the useris speaking with the clothes and the background stored in advance.

The synthesizing may also be effected for not only the mouth but alsofor the whole of the face, including the movement of eyes and skin.Also, it is possible to transmit to the other party not a tiredexpression but synthesizing of a healthy, bright expression with thecurrent movement of the mouth, or an expression involving movement ofthe eyes and skin, for example, and thereby render a pleasant sensationto the other party in the communication.

Identification of the user may be performed by comparing the obtainedimage with a stored image, and it may also be performed by providing auser ID to each user to be input.

As described above, by changing the images obtained by the camera intoimages obtained by the user in advance using the system, andsynthesizing these images to form the image of the user to betransmitted, according to the state of the user, it is possible to avoidtransmitting of images which the user does not wish to transmit, and totransmit the expression of the user clearly to the other party. It isthus possible to maintain privacy and to transmit an image which is notimpolite to the other party.

A description will now be given relating to the basic method of imageprocessing.

The image processing apparatus converts the input image into a binaryimage by binarization. The binarization sets a value of 1 if thebrightness is a fixed value (threshold value) or more, and sets 0otherwise. The threshold value can be determined by the followingmethods.

(1) In a fixed threshold value method, the threshold value is set inadvance. This method can be used if the variations in the image is lessand the image has at least sufficient contrast.

(2) A p tile method is effective if the ratio of “1” portions within theimage is known in advance. Normally, a brightness histogram is taken,and the threshold value is set to a brightness such that the cumulativedistribution is p percent. Even if the histogram is not produced inadvance, it is possible to obtain a threshold value which givesapproximately p percent by binarizing using a suitable threshold value,by repeating obtaining of the ratio of “1”.

(3) In a mode method, if the brightness histogram is bimodal, the valueof the brightness in the troughs is taken as the threshold value. Therealso exists a variable threshold value method in which a fixed thresholdvalue is not applied to the whole image, but the image is divided intoregions, and threshold values are determined by the mode method for thedivided regions.

(4) In a floating binarization and moving average threshold valuemethod, if the amplitude of the brightness signal is attenuated and thedelayed signal is set as the threshold value, it is possible to detectsudden variations in brightness. A similar method which is also used isa moving average method in which the average brightness over a fixedperiod is set as the threshold value.

FIG. 2 shows the principle of pattern matching whereby the location of aparticular pattern is detected. The object image is made into a binaryimage, and the region to be detected is set as “1”. This region is setat a suitable position on the image, and the degree of overlap with theobject image is examined in a mismatch detection block 10. A pluralityof positions are set in a position setting block 11. If the overlap isdetected in the positions, counting is performed in a counting block 12.A count value is detected in a minimum value detection block 13. Theposition of the pattern is obtained when the degree of overlap ismaximum and the degree of mismatch is small. If the degree of mismatchis not small in any position then such a pattern is deemed not to bepresent.

If the two-dimensional pattern of the object is not constant, simplepattern matching is insufficient. Various features must be extractedfrom the image and determination made based on a combination thereof. Afeature obtained from one portion of the image is called a localfeature, and subsequent processing is often performed based on this.

To obtain what is shown in the image, it is necessary to extractfeatures from the image, produce a description of the object and matchthis with a previously formed model. Features include points, lines andregions and a number of levels. That is, features can be extracted fromonly a small local region, or they can reflect the state of a largerregion. Further, features can be expressed in a layered manner such thatpoints are joined together to form lines, and regions enclosed by linesare obtained. For each feature, which level is suitable depends on thepurpose.

Space differential is often used to perform extraction from a gradedimage or an edge which is part of the boundary between two regions withdifferent brightnesses. In order to obtain the difference of thebrightness at a certain point and the direction thereof, the gradient ofthe brightness f(x, y) should be calculated. If f(x, y) is a continuousimage, the gradient grad [f(x, y)] is a two-dimensional vector indicatedby formula (1).

grad[f(x,y)]=(∂f(x,y)/∂x, ∂f(x,y)/∂y)  (1)

If the absolute value is large, there is an edge and the perpendiculardirection is taken as the gradient. However, if the width of the line islarge to some extent then if the direction has two opposite gradients,this can be taken to be the boundary of the line. Further, if the widthof the line is small, the Laplacian of formula (2) can be used.

Δ² f(x,y)=∂² f(x,y)/∂x ²+∂² f(x,y)/∂y ²  (2)

If the value of the Laplacian is large, this point is darker than therest, and if it is negative then it is brighter than the rest.

To extract a single line by joining edges or line elements, the lineelements are obtained, the image is divided into small regions of 4×4pixels, and the largest differential values from the edge points (edgeelements) within each small region are left and the others are erased.The direction of an edge is obtained by examining the brightness in thevicinity of each edge point. If the edge direction within adjacent smallregions is approximately the same, the two edge points are joined.Points which are not connected at all are deemed to be noise and areerased. At each stage of this process, only the vicinity of the image isused, and furthermore processes for each point are independent of theprocesses for other points. Such processing is known as localprocessing, and it is possible to perform processing in parallel foreach point.

To detect a line by performing local processing a number of times, thereare methods which use probability as described below. First, bydetecting edge points, the strength of the edge point (absolute value ofdifferential) and direction are obtained. An initial value for theprobability that each edge point is on a line is determined based on thestrengths of the edge points. Next, the probability for each edge pointis changed by the strength and direction of neighboring edge points. Ifa certain edge point and a different edge point in the vicinity have thesame direction, the probability of the edge point is increased. If thedirections differ greatly, it is reduced. By performing such correctiona number of times, gradually noise is erased and the probability of anedge point being on a line approaches 1. The process is known as therelaxation method.

In the parametric space method, a straight line is represented by ρ=xcos θ+y sin θ. If the coordinates (x_(i), y_(i)) of the resultingboundary points are converted to a θ-ρ plane, a curved line in thisplane, indicated by ρ=x_(i) cos θ+y_(i) sin θ is obtained. If all of theboundary points within a scene are converted, all points on the samestraight line pass through one point in the θ-ρ plane. Thus if a point(θ₀, ρ₀) through which a large number of curves passes is obtained, theequation for the straight line can be obtained. If the direction of theboundary points is known, the straight line can be determined from ahistogram relating to ρ. The parameter method requires a specialhistogram to be produced with the same number of dimensions as thenumber of unknown parameters, and is therefore difficult to apply to acurved line. However, it is effective if there are fixed constraints onthe parameters included in the equation for the curved line, and thedimensions of the parameter space can be reduced to two or less.

The tracking method is applicable to obtain a general curve. Thetracking method is a process whereby lines are produced connectingadjacent edge points. The following procedure is required to performtracking.

(1) Detect the point at which tracking is to begin.

(2) Estimate the position, and in some cases the direction, of theadjacent edge point.

(3) Detect an edge point based on this estimation.

(4) Determine whether to continue tracking or to finish tracking.

Tracking involves serial processing, in contrast to local processing.Therefore if a single edge point is to be detected, this process dependson the result of previous processing. Therefore the obtained result willbe different depending on where in the image the processing starts.However, with serial processing, it is probable that processing can beperformed effectively by using previously obtained informationeffectively.

A method for extracting uniform regions is known as the region method,and include the following two processes.

(1) In a region method using merging, firstly, a large number of smallregions are produced, and similar regions are merged.

(2) In a region method using division, starting with a single region,this is divided into regions with different characteristics.

The region method using merging initially obtains small regions in whichthe brightness is approximately the same, known as base regions. Regionsare gradually merged based on the difference between the brightnesses oftwo adjacent regions, the length of the boundary and the shape of theregions. This process is repeated until no more regions can be merged.

The method using division has the important problem of determining onwhat basis the image should be divided. Dividing a color image based onthe histograms of various properties is effected as follows. Firstly,the red state, green state, blue state, brightness, color phase,saturation, and the Y, I and Q color television signals are detected foreach pixel. Nine types of histogram are produced from the parameters,and the parameters which form conspicuous peaks and troughs areselected, and divisions are made to the left and right of troughs, whichare used as boundaries. The image is divided depending on whichproperties the pixels have. The various histograms are then produced forthe divided regions, and these are once again divided more finely. Thisprocess is repeated until it is no longer possible to make divisions.

The region method using division is effective if it is possible to usenot simply brightness, but a relatively large amount of information, asin this example. However, it is not necessary to use as many as 9 types,as in this example.

By using the above methods, it is possible, after obtaining the imagedata of the user, to determine whether or not the user has blinked, andto calculate the bloodshot degree of the eyeballs of the user. In thepresent embodiment, the eyeballs of the user is monitored using a CCDcamera, but the direction of gaze of the eyeballs, or the reaction ofthe pupils may also be observed using a goggle-type head-up displaydevice.

In the embodiment, it is shown in a case wherein the clothes and thebackground are changed. However, if it is determined that the user isextremely tired, then it is possible to change the actual expression ofthe user by image processing, and to transmit to the other informationprocessing system the image data which synthesizes a bright, healthyface from a time at which the user was in good health. Further, at thistime is also advantageous to control the tone or pitch of the voiceslightly such that if the pitch of the voice is liable to be lowered,for example if the user is extremely tired, a bright voice is formed.

The information processing system according to the present invention isprovided with a camera and a microphone, and makes it possible toprotect the privacy of parties involved in communication when performingdata communication by image data, by synthesizing image data obtained bythe camera with image data stored in advance, either based on adetermination made from the image data, or at the will of thetransmitting party in the communication, and thus making the clothes,hair, background, make-up etc. different from reality, and transmittingthese data to the other party in the communication.

What is claimed is:
 1. A videophone comprising: a camera for obtainingimage data of a user of said videophone; a microphone for obtainingvoice data of the user; means for detecting a degree of tiredness of theuser; means for modifying said image data by image processing; means fortransmitting the image-processed image data and the voice data toanother videophone remote from said videophone; and a goggle-typehead-up display device for displaying an image from another videophoneto be viewed by said user.
 2. A videophone according to claim 1 whereinsaid image data includes background data.
 3. The videophone according toclaim 1 further comprising means for modifying the voice data inaccordance with said degree of tiredness.
 4. The videophone according toclaim 1 wherein said image data is modified in accordance with saiddegree of tiredness.
 5. The videophone according to claim 1 wherein saidtiredness is determined based on detecting the number of blinks of aneyeball of the user or a bloodshot degree.
 6. The videophone accordingto claim 1 wherein said tiredness is determined based on detecting abloodshot degree of an eyeball of the user.
 7. A videophone comprising;a camera for obtaining image data of a user of said videophone includingbackground data; a microphone for obtaining voice data of the user;means for detecting a degree of tiredness of the user; means for storingdesired background data; means for changing the background data for thedesired background data by image processing; means for transmitting theimage-processed data and the voice data to another videophone remotefrom said videophone; and a goggle-type head-up display device fordisplaying an image from another videophone to be viewed by said user.8. The videophone according to claim 7 further comprising means formodifying the voice data in accordance with said degree of tiredness. 9.The videophone according to claim 7 wherein said background data ismodified in accordance with said degree of tiredness.
 10. The videophoneaccording to claim 7 wherein said tiredness is determined based ondetecting the number of blinks of an eyeball of the user or a bloodshotdegree.
 11. The videophone according to claim 7 wherein said tirednessis determined based on detecting a bloodshot degree of an eyeball of theuser.
 12. A videophone comprising: a camera for obtaining image data ofa user of said videophone; a microphone for obtaining voice data of theuser; a processor for modifying at least the image data; means fordetecting a degree of tiredness of the user; transferring means fortransmitting the modified image data and the voice data to anotherinstrument remote from said videophone; and a goggle-type head-updisplay device for displaying an image from said instrument to be viewedby said user.
 13. A videophone according to claim 12 wherein said imagedata includes background data.
 14. The videophone according to claim 12further comprising means for modifying the voice data in accordance withsaid degree of tiredness.
 15. The videophone according to claim 12wherein said image data is modified in accordance with said degree oftiredness.
 16. The videophone according to claim 12 wherein saidtiredness is determined based on detecting the number of blinks of aneyeball of the user or a bloodshot degree.
 17. The videophone accordingto claim 12 wherein said tiredness is determined based on detecting abloodshot degree of an eyeball of the user.
 18. A videophone comprising:a camera for obtaining image data of a user of said videophone; amicrophone for obtaining voice data of the user; a converter fordigitizing said image data and said voice data of the user; a processorfor modifying at least the digitized image data; means for detecting adegree of tiredness of the user; transferring means for transmitting themodified image data and the voice data to another instrument remote fromsaid videophone; and a goggle-type head-up display device for displayingan image from said instrument to be viewed by said user.
 19. Avideophone according to claim 18 wherein said image data includesbackground data.
 20. The videophone according to claim 18 furthercomprising means for modifying the voice data in accordance with saiddegree of tiredness.
 21. The videophone according to claim 18 whereinsaid digitized image data is modified in accordance with said degree oftiredness.
 22. The videophone according to claim 18 wherein saidtiredness is determined based on detecting the number of blinks of aneyeball of the user or a bloodshot degree.
 23. The videophone accordingto claim 18 wherein said tiredness is determined based on detecting abloodshot degree of an eyeball of the user.
 24. A videophone comprising:a camera for obtaining image data of a user of said videophone; amicrophone for obtaining voice data of the user; a converter fordigitizing said image data and said voice data of the user; a memory forstoring the digitized image data and the digitized voice data; aprocessor for modifying at least the digitized image data; means fordetecting a degree of tiredness of the user; transferring means fortransmitting the modified image data and the voice data to anotherinstrument remote from said videophone; and a goggle-type head-updisplay device for displaying an image from said instrument to be viewedby said user.
 25. A videophone according to claim 24 wherein said imagedata includes background data.
 26. The videophone according to claim 24further comprising means for modifying the voice data in accordance withsaid degree of tiredness.
 27. The videophone according to claim 24wherein said digitized image data is modified in accordance with saiddegree of tiredness.
 28. The videophone according to claim 24 whereinsaid tiredness is determined based on detecting the number of blinks ofan eyeball of the user or a bloodshot degree.
 29. The videophoneaccording to claim 24 wherein said tiredness is determined based ondetecting a bloodshot degree of an eyeball of the user.